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ABSTRACT 

Under pressures of increased demands for public 
accountability and decreased resources, the State University System 
of Florida has responded by fusing existing program review procedures 
with accountability reporting. The State University System began 
program review in the mid-1970s and found it most valuable internally 
for the individual universities and their governing boards 
particularly in advocating for faculty and students needs. By 
contrast, accountability reporting for the entire system began in 
1991 at the behest of the state legislature with an adversarial 
thrust stemming from a core sense that faculty were not working hard 
enough. An original 50 accountability measures were eventually 
streamlined to 9 objectives and a resulting merger of review and 
accountability reporting has resulted in less paperwork and reporting 
and the opportunity to merge quantitative outcome indicators with 
process-oriented qualitative measures of excellence leading to an 
enfranchisement of all stakeholders. The program review has shifted 
emphasis from one of enhancing quality and programmatic offerings in 
a political climate favorable to higher education to quality 
assurance under diminishing resources, demands for increased access, 
and increased societal criticism of higher education and to ways to 
bring about these changes while protecting the integrity of academic 
programs. Overall, program review offers the significant advantage of 
addres^in^ .iccountabi 1 i ty concerns in a responsible manner that takes 
into account the complexities of academic programs. An appendix lists 
the original 50 accountability measures. (JB) 
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RESTORING CREDIBILITY: 
THE NEXUS BETWEEN ACCOUNTABILITY AND PROGRAM REVIEW 



INTRODUCTION 

Accountability, a national and even international 
phenomenon, has become synonymous with the process of attempting 
to ensure that universities are wise users of tax dollars. It 
has gone hand in hand with a decrease both in resources and in a 
general trust in the higher education system. Program review 
predates the recent interest in accountability, and is itself 
enjoying a resurgence of interest. It is natural that states 
would look toward utilizing already existing program review 
processes for accountability purposes. This paper will discuss 
the fusion of program review procedures and accountability 
reporting, the emergence of 50 measures which were initially used 
for accountability reporting in a major State University System, 
and the evolutive streamlining of both the procedures and the 
criteria for accountability reporting. 

There is no indication that accountability is anything other 
than here to stay, and the extent to which funding to higher 
education may, over time, be restored or increased will depend in 
part on how universities and systems respond to this new level of 
scrutiny. Those universities which take a proactive, leadership 
role will likely fare better than those which are being dragged 
ill the way to the same inevitable table. 

More importantly, the public's long-standing trust in 
education has seen an erosion unequaled in recent history. It 
may be through accountability and the measures subordinated under 
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that name that this trust will be restored. Academic program 
reviews, already in existence in many states, could provide a 
powerful tool for certain facets of accountability, taking it 
beyond purely quantitative data to a richer portrait of the 
performance and effectiveness of an institution's academic units. 

PROGRAM REVIEW: Background 

The State University System of Florida's (SUS) reliance on 
academic program review is longstanding. The SUS began exploring 
the possibilities of conducting program reviews with the hiring 
of a new Vice Chancellor for Academic Affairs in the mid 1970 's. M 
Subsequently, the first cyclic program reviews were initiated in 
1975 with the intention that, eventually, all academic programs 
would be reviewed, at which time the cycle would begin anew. 
This has occurred. A? 1 disciplines in the SUS have enjoyed this 
scrutiny, with the greater majority having been reviewed a second 
or even a third time via a peer review process that sees ten to 
twelve reviews in progress at any given time within the nine 
universities comprising the SUS. In 1983, program review 
activities were formally placed into law. 

It is important to note several important characteristics 
which distinguish the program review process as it is conceived, 
"marketed," and implemented in the SUS from other review 
procedures. First, however far-reaching the distribution of 
printed program review reports, it was clear from the outset that 
their value was highest internally, that is, within the confines 
of the various working relationships of the individual 
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universities and their single governing board. As such, program 
reviews served and continue to serve in the dual capacity of a 
quality control mechanism, but, as importantly, as a major 
informational source with respect to existing activities and, 
significantly, the aspirations of the universities to create new 
academic degree programs, institutes and centers, organizational 
units including departments, schools, and colleges; as well as 
termination of activities and, finally, other curricular issues 
which, from time to time, came before the purview of the 
governing body. The distinction between informational and purely 
evaluative motive is important to understand in this context. 

Secondly, and due in large part to its peer review 
methodology, program review was marketed and understood to be, if 
swaying in any one direction, in the direction of an advocacy 
position, particularly for faculty and students. Program reviews 
were seen as a forum for making known to the central 
administration of individual universities as well as to the 
governing board the various resource needs in the form of more 
faculty, equipment, dollars for assitantships and faculty 
development. Moreover, the process served to demonstrate the 
performance of relevant research and service, both activities of 
which have been difficult to describe to constituents external to 
the academic community, 

ACCOUNTABILITY : Background 

By contrast, accountability reporting for the System began 
in 1991 not at the behest of the governing board itself but, 



rather, directly from the legislature as it attempted to 
scrutinize the use of public dollars by public agencies. Unlike 
the program review process, accountability has, from the outset, 
had an almost adversarial thrust as opposed to the advocacy 
positions of program reviews. This has stemmed from a core 
notion of accountability with respect to higher education in 
Florida, namely, that faculty are not working hard enough. 

In 1994, accountability and program review were merged into 
a single conceptual framework to be driven by the Master Plan 
goals of the SUS (the SUS Master Plan establishes five-year goals 
for the system of ten universities) • The fusion of program 
review and accountability must be preceded by a thoughtful 
conceptual analysis of the two activities because it entails the 
merging of a process which has primarily been a tool for quality 
improvement initiated by academicians (i,e, program review), and 
a process which is primarily a tool for evaluating the use of tax 
dollars initiated by the public and legislators (i,e, 
accountability) . The credibility of both program review and 
accountability in the eyes of academia, legislators, and tax 
payers must be maintained. 

After an initial year in which accountability measures 
listed in statute were reported on, members of the academic 
community expanded the process to encompass 50 accountability 
measures which were to provide the legislature with an assurance 
that the complexity of activities on SUS campuses were effective 
and accountable (a summary of the 50 measures appears in the 
Appendix) . This proved to be unworkable and, in general, 
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ineffective for the simple reason that the 50 measures were 
predicated on the notion of "telling our story" to the 
legislature, when, in fact, the legislature was not interested in 
the story of academia; rather, the focus of the legislature was 
on a series of clearly articulated "heartburn" issues which were 
physically, politically, and economically distressing to both the 
legislative body as well as to the individual universities and 
the SUS as a whole. 

Consequently, the 50 measures have now evolved toward a more 
streamlined association to the original intent of the 
legislature. It is hoped that this streamlining will make the 
entire process more negotiable. The final accountability 
objectives which have ber i settled on are listed below. 



One: Increase or maintain a high level of undergraduate 

teaching productivity by state funded ranked faculty, 
while maintaining or reducing average class size. 

Two: Meet planned enrollment. 

Three: Increase access to state universities. 

Four: Incre ise undergraduate retention and graduation rates 

Five: Streamline progression to baccalaureate degree. 

Six: Increase undergraduate degree productivity. 

Seven: Increase Research productivity. 

Eight: Increase Service Productivity 

Nine: Increase Institutional Support from External Sources 



The endeavors of program review and accountability, refined 
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over time, have provided the SUS with the experiential and 
conceptual framework with which to merge the processes into a 
seamless strategic planning and evaluative procedure which has 
resulted in less and not more paperwork and reporting, and the 
opportunity to merge quantitative outcome indicators with 
process-oriented qualitative measures of excellence. This merger 
has been instrumental in the enfranchisement of all stakeholder, 

from legislators to faculty. 

The SUS' procedures for program review, its accountability 
measures, and its organizational infrastructure are capable of 

informing similar strategies both at system as well as at 

institutional levels . 

Purposes of Program Review 

Program reviews became widely used as quality assurance 
activities in the United States beginning in the 1970 's (Bogue 
and Saunders, 1992) . The foll<"wing examples of definitions of 
program review provide a sense of the general purposes of such 
reviews: "... a comprehensive evaluation of a curriculum leading 
to a degree. This review will ordinarily involve the acquisition 
of historic, current, and projective data on program purpose, 
resources used and needed, and an evaluation of performance" 
(Bogue and Saunders, 1992, p. 138); "...evaluation of programs 
already in existence" with the objectives being to generally 
focus on program improvement and efficiency (Barak, 1982, p. 93- 
94) ; and "the process of defining, collecting, and analyzing 
information about an existing program or noninstructional unit to 



arrive at a judgment about the continuation, modification, 
enhancement, or termination of the program or unit'- (Conrad and 
Wilson, 1985, p. 10) . 

A study conducted by Moon-Hee Lee (1991) , on t:he purposes of 
program review in the 39 states which conduct statewide program 
reviews, found that the purpose most frequently identified was 
that of assessing and enhancing program quality • Although 
increasing efficiency and effectiveness of resources, providing 
optimum service and access, and eliminating costly duplication of 
programs were cited as purposes of program review, they were 
identified much less frequently than the purpose of assessing and 
enhancing program quality. The program reviews in the SUS 
reflected these national trends. While efficiency and avoiding 
duplication were traditional components of SUS program reviews, 
the main purpose was to evaluate and enhance the quality of the 
programs . 

Program Review Checklist 

The program reviews in the SUS employ both quantitative and 
qualitative evaluation procedures to capture the rich complexity 
of academic programs and reflect that complexity in the 
recommendations which result from program reviews. The SUS 
utilizes a peer review model with external consultants who are 
experts in the disciplines under review. Prior to the site 
visits, the consultants are provided a generic outline of 
programmatic points to address. The points on the outline are 
the result of 20 years of SUS experience with program reviews. 
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and have been found to yield good coverage of all major 
components of academic programs. The major headings on the 
Checklist comprise four general sections: Program, Faculty, 
Students, and Facilities and Resources. The Checklist appears in 
Figure 1. 

Shifting the Focus of Program Review to Incorporate 
Accountability Concerns 

As the public demands for accountability in higher education 
have increased, and the legislative requirements for 
accountability have become urgent, the SUS program reviews have 
begun a shift in focus. 

The context in which program reviews in the SUS are 
conducted has changed from one of enhancing quality and 
increasing programmatic offerings, in a political and economic 
atmosphere that was relatively favorable to higher education in 
the 1970 's and 1980 's, to a context of quality assurance in an 
atmosphere of diminishing resources, demands for increased 
access, and increased societal criticism of higher education. An 
underlying issue for program review today is how to enhance 
quality and be accountable to the public. In Florida, and across 
the nation, it has become evident that the public's most pressing 
concerns are not the traditional concerns of quality as 
understood by higher education. They are less concerned with the 
quality of the faculty, curricula, equipment, and other 
resources. Their main concern is that universities should focus 
on undergraduate teaching and learning. "Their issue is whether 
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FIGURE 1 



Program Review Checklist 


REMARKS 








Depth, breadth, and currency of curriculum 


- 


Programs properly sequenced 




Appropriate number of credit hours to BA/BS degrees 




Adequate availability of courses 


i 


Appropriate conimunir\' college articulation 




Appropriateness of Limited Access, if applicable 




Appropriate use of advisory boards 




Linkage with relevant institutes and centers 




Accreditation status 




Distance learning activities 




Pending programmatic modifications 





Summary/Overview 
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Program Review Checklist 




REMARKS 




• ' .'STUDENTS V : :. ' ' " 






Appropriate admission standards 






Appropriate number of majors 






Appropriate retention and graduation rates 






Passing rates on licensure examinations, if applicable 






Adequate number of teaching assistantships 






Adequate teaching assistant stipends 






Graduate Placement; Graduate/employer satisfaction 






Provision of clubs, societies, etc. in discipline 






Adequate advising services 






Appropriate distribution by race and gender 






S u m ma ry/CKerview 
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Program Review Checklist 



REMARKS 









Appropriate distribution by ag'^, rank, gender, and race 




- 


Quality of departmental leadership 






Quality of teaching 






Quality of scholarship and publication 






Appropriate contributions by Eminent Scholars 






Internal cooperation 






External linkages 






Budgetary provision for faculty development 






Salaries 







Summarv/Overview 
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Program Review Checklist 



REMARKS 



FACILITIES AND RESOURCES 






Faculty office space 




- 


1 Teaching assistant office space 






1 Classroom space 






1 Laboratory* space 






1 Provision of studenl^facult)' commons room 






IDepanmental Conference Room 






1 Adequate clinical facilities 






1 Equipment 






Adequate office expense budget 






j Appropriate number of support personnel 
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the institutions are doing the right things, not whether they are 
in the traditional sense doing things right." (NCHEMS, 1994) 
An important role for program review has become one of 
identifying ways to bring about demanded changes in a manner 
that, where possible, protects the integrity of academic 
programs. These demands, usually from the State legislature, are 
reflected in the accountability measures discussed earlier. They 
include decreasing the credit hours required for degree programs, 
increasing the undergraduate teaching productivity of faculty, 
and increasing access to more students, withe^ut proportional 
increases in resources. 

One means of visualizing the interaction between program 
review and accountability is through the framework presented in 
Figure 2. Accountability issues which focus on universitywide 
data, or which can be explored through readily available 
quantitative measures are, at least initially, generally 
addressed directly through the accountability process. However, 
when both quantitative and/or qualitative measures, and 
programmatic information are called for, or th-^ quantitative 
measures in the accountability process reveal a problem that 
could be illuminated by qualitative review by experts in a 
discipline, the issue is taken up in the program review process. 
'For example, an accountability measure might reveal that 
licensure pass rates in a particular discipline are low. Program 
review could explore the program-specific reasons for the low 
scores, recommend a course of action, and legitimize the concern 
and recommendations in the eyes of the faculty, through the 



13 



ERIC 15 



FIGURE 2 



ACCOUNTABILITY 
CONCERN 




IS QUANTITATIVE 
DATA SUFFICIENT ? 



YES 




NO 



ACCOUNTABILITY 
MEASURE & REPORT 



ADDRESS IN 
PROGRAM REVIEW 



REVEALS AN AREA 
WHICH NEEDS 
IMPROVEMENT 



DISTILL FINDINGS AND 
RECOMMENDATIONS; 
REPORT FOR 
ACCOUNTABILITY 



IF NECESSARY, FEED BACK 
INTO PROGRAM REVIEW 
FOR RECOMMENDATIONS 
ON HOW TO IMPROVE 
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involvement of respected peer reviewers in their discipline. 

The SUS program review checklist, because of its 
comprehensiveness, is still a good tool for program reviews, and 
the SUS v/ill continue to address all points on the checklist 
because they provide a holistic portrait of the program, rather 
than an unbalanced view that is only responsive to the 
vicissitudes of legislative concerns on accountability. However, 
certain items on the checklist will receive greater attention, or 
a different focus, than had been the case previously. Examples 
of areas of increased emphasis in program review, resulting from 
accountability concerns, follow. 

Faculty Teaching Productivity, For each program under 
review, central questions include: How can faculty teaching 
productivity, particularly at the undergraduate level, be 
increased? Is the level of teaching appropriate to the mission 
of the university? i.e. universities in the system whose primary 
mission is teaching would have different expectations for 
undergraduate teaching productivity than the Research I 
universities (according to the Carnegie classification) . Are 
there faculty who are not very productive in research who could 
teach more courses? The universities in the SUS are increasingly 
providing variable distribution of effort for teaching and 
research, whereby those whose strengths lie in research may teach 
fewer courses than those whose strengths lie in teaching. The 
SUS is also undergoing a major review of tenure criteria, 
providing more emphasis on teaching than they had before. The 
move to increase faculty teaching must necessarily go hand in 
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hand with changes in the reward structure which recognizes 
teaching. 

Student Retention and Graduation Rates, Program reviews 
will provide greater scrutiny of retention and graduation rates 
for programs, and attempt to identify strategies for improving 
low rates. 

Student Advising and Other Academic Support. The efficacy 
of student advising in decreasing excess credit hours taken by 
students (i.e. credit hours that are not required for 
graduation) , and the provision of adequate academic support to 
promote student retention and graduation will be examined • 

Credit Hours to Degree and Beyond Degree. Program review 
will examine how these credit hours may be decreased where 
necessary, with minimal impact on the academic quality of the 
program. Programs which can reasonably be offered within 120 
semester hours will be decreased to that level, and those which 
must exceed 120 will be justified, in keeping with a recent 
legislative requirement • 

Student Performance on Licensure Excuninations . Prominent 
among the outcomes measures in which legislators have expressed 
an interest are passage rates on licensure examinations. Program 
reviews are sensitive to this, and provide information on how 
pass rates in the program reviewed compare to national averages. 
In the absence of such comparisons and other considerations 
specific to each discipline, the pass rates alone can be 
misleading. Where improvements are needed, the reviews also 
suggest means of improving the rates. 
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Faculty Research and Service. Increasingly , the question on 
program reviews will be whether faculty are conducting 
significant research and whether faculty service rendered is 
indeed an appropriate application of disciplinary expertise to 
benefit the university community or the local community. In 
other words, do the activities in these two areas merit the 
expenditure of State funds? 

External Resources. The health of academic programs in some 
universities increasingly will depend on the ability of the 
programs to generate external resources. Program reviews, where 
appropriate, will evaluate the faculties' competitiveness in 
garnering external funds, and may recommend avenues for 
increasing external resources. 

Viability of Programs and Selectivity in Initiating New 
Programs. The termination of academic programs has traditionally 
not been a major part of the agenda for program reviews in the 
SUS. In part, this was designed to allay faculty fears regarding 
reviews, and encourage them to be full participants in the 
reviews and the implementation of recommendations. As resources 
diminish and the demands for accountability from constituencies 
external to higher education become more strident, program 
productivity is coming under increased scrutiny. Program reviews 
will need to address, to a greater extent than in the past, the 
viability of programs identified, through data analysis, as 
having "low productivity." In regard to the initiation of new 
degree programs, program reviews have always played an important 
role. Now that greater selectivity needs to be exerted in 
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initiating new programs, the evaluation of a university's ability 
to mount a program of high quality will become even more 
rigorous. 

IV. Conclusion 

Much of the public debate on higher education today centers 
around accountability. How well institutions of higher education 
address the concerns of the public through accountability will, 
to a great extent, determine future public support. Program 
reviews, which many institutions already have in place, and 
several institutions are now initiating, offer a means of 
addressing some of the concerns in regard to accountability 
without more duplicative activities. 

Program review offers the significant advantage of 
addressing accountability concerns in a responsible manner that 
takes into account the complexities of academic programs. This 
is particularly important to preserve the integrity of higher 
education within the context of accountability, because 
accountability issues are often framed by those not familiar with 
higher education, and tend to use simple, quantitative measures. 
While there is little doubt that higher education will need to 
take difficult steps to address the accountability concerns 
voiced by the public and legislators, we must ensure that those 
steps will have the intended outcomes, and benefit students and 
society both in the short and long term. Program reviews are a 
tool which can bring quantitative and qualitative measures, and 
some of the best minds in each academic discipline, to bear upon 
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this challenging task. 
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Student Surveys; Quality of Instruction. 



Departmental Initiatives for Improving Teaching 



Provisions for Faculty Development. 



Results of Attempts at Accreditation. 



Licensure Passing Rates. 



Employer Surveys: Satisfaction with Graduates. 



Programs to Promote Computer Literacy. 



Increase of First Time In Collese Enrollments. 



Increase Number of Minoritv Scholars. 



Increase Alternative admits Retention and Graduation. 



Increase Minority and Women Enrollments If Under-represented. 



Keep Degree Programs to 120-128 Credit hours to Degree. 



Provide Courses Based on Demand. 



Twelve Contact Hour Production. 



Graduation Rates of Athletes the Same as for All Students. 



Meet Planned Enrollment Levels. 



Increase % of Full-time Students Graduating in Four Years. 



Reduce Number of Lim.ited Access Decree Programs. 



Increase % of Full-time CC Transfers Graduating in Two Years. 



Increase Retention Rates. 



Number and Type of Retention programs. 



Provide Stable Source of State Fundinr to SUS. 



Receive Funding to Improve Faculty Salaries by 107o. 



Establish Differential Fees for Purposes of Flexibility. 



Review T' iorities Through Resource and Productivity Model. 
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Planning Directive One: 

Improve the Quality of 
Undergraduate Education 



Planning Directive Two: 

Provide Adequate Access to 
Undergraduate Education 



Planning Directive Three: 

Establish a Stable, Reliable 
Source of State Funding 
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39 
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47 



48 
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Develop Creative and Cost Saving Programs. 



Achieve SUS Average of 27 FTE in Graduate Programs. 



Increase Distance Education Activities 



Improve Computer & Telecommunications Systems. 



Increase Number of Minority Law Students. 



Increase Interest in Teaching as a Profession. 



Increase Faculty Involved in Public Schools. 



Increase School District Personnel Involved in SUS. 



Share In-service Training With Public Schools 



Research Agenda in Cooperation With Public Schools. 



Improve Student Assessment in Public Schools. 



Gear Teacher Training Programs Toward Blueprint 2000. 



Increase Primary Care Physicians from Medical Schools. 



Increase Production of Nurses. 



Increase Production of Allied Health Graduates. 



Produce Quality Scholaiship in Areas Critical to Florida. 



Provide Public Service in Critical Areas. 



Increase Contract and Grant Income. 



Increase Private Funding. 



Compete for Fed. Defense Conversion Tech. Awards. 



Increase Library Collections. 



Expand Use of Technology in SUS Libraries. 



Increase Library Funding to Comparable Institutions., 



Survey: Student Satisfaction with Ad ising. 



Increase Usefulness of Technologically-based Advising. 
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Planning Directive Four: 

Develop and Implement Creative 
and Innovative Cost-saving Pro- 
grams to Increase Efficiency with- 
out Sacrificing Quality 

Planning Directive Five: 

Solve Critical Problems in a 
Rapidly Growing State. 



Planning Directive Six: 

Forge Public/Priate Partnerships to 
Help Achieve State Goals 



Planning Directive Seven: 

Improve the Quality of the Librar- 
ies 



Planning Directive Eight: 

Provide Quality Student Advising 



